Decision-support application and system for medical differential-diagnosis and treatment using a question-answering system

ABSTRACT

A decision-support system for medical diagnosis and treatment comprises software modules embodied on a computer readable medium, and the software modules comprise an input/output module and a question-answering module. The method receives patient case information using the input/output module, and generates a medical diagnosis or treatment query based on the patient case information and also generates a plurality of medical diagnosis or treatment answers for the query using the question-answering module. The method also calculates numerical values for multiple medical evidence dimensions from medical evidence sources for each of the answers using the question-answering module and also calculates a corresponding confidence value for each of the answers based on the numerical value of each evidence dimension using the question-answering module. The method further outputs the medical diagnosis or treatment answers, the corresponding confidence values, and the numerical values of each medical evidence dimension for one or more selected medical diagnosis or treatment answers using the input/output module.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present invention claims the benefit under 35 U.S.C. §120 as aContinuation of presently pending U.S. patent application Ser. No.13/077,480 filed on Mar. 31, 2011, the entire teachings of which areincorporated herein by reference, which also claims priority to U.S.Provisional Application Ser. No. 61/450,273 filed on Mar. 8, 2011, thecomplete disclosure of which, in its entirety, is herein incorporated byreference.

BACKGROUND

The embodiments herein relate to using a question-answering system tosupport a human expert in problem solving in a particular domain, andmore specifically to a decision-support application and system forproblem solving using a question-answering system.

Decision-support systems exist in many different industries where humanexperts require assistance in retrieving and analyzing information. Anexample that will be used throughout this application is a diagnosissystem employed in the health care industry.

Diagnosis systems can be classified into systems that use structuredknowledge, systems that use unstructured knowledge, and systems that useclinical decision formulas, rules, trees, or algorithms. The earliestdiagnosis systems used structured knowledge or classical, manuallyconstructed knowledge bases. The Internist-I system developed in the1970s uses disease-finding relations and disease-disease relations, withassociated numbers such as sensitivity, the fraction of patients with adisease who have finding (Myers, J. D. The background of INTERNIST-I andQMR. In Proceedings of ACM Conference on History of Medical Informatics(1987), 195-197).

The MYCIN system for diagnosing infectious diseases, also developed inthe 1970s, uses structured knowledge in the form of production rules,stating that if certain facts are true, then one can conclude certainother facts with a given certainty factor (Buchanan, B. G. andShortliffe, E. H. (Eds.) Rule-Based Expert Systems: The MYCINExperiments of the Stanford Heuristic Programming Project.Addison-Wesley, Reading, Mass., 1984). DXplain, developed starting inthe 1980s, uses structured knowledge similar to that of Internist-I, butadds a hierarchical lexicon of findings (Barnett, G. O., Cimino, J. J.,Hupp, J. A., Hoffer, E. P. DXplain: An evolving diagnosticdecision-support system. JAMA 258, 1 (1987), 67-74).

Iliad, developed starting in the 1990s, adds more sophisticatedprobabilistic reasoning. Each disease has an associated a prioriprobability of the disease (in the population for which Iliad wasdesigned), and a list of findings, along with the fraction of patientswith the disease who have the finding (sensitivity), and the fraction ofpatients without the disease who have the finding (1-specificity)(Warner, H. R., Haug, P., Bouhaddou, O., Lincoln, M., Warner, H.,Sorenson, D., Williamson, J. W. and Fan, C. ILIAD as an expertconsultant to teach differential diagnosis. In Proc. Annu. Symp. Comput.Appl. Med. Care. (1988), 371-376). DiagnosisPro(http://en.diagnosispro.com) is a structured knowledge base that can bequeried and browsed online.

In 2000, diagnosis systems using unstructured knowledge started toappear. These systems use some structuring of knowledge. For example,entities such as findings and disorders may be tagged in documents tofacilitate retrieval. ISABEL uses Autonomy information retrievalsoftware and a database of medical textbooks to retrieve appropriatediagnoses given input findings (Ramnarayan, P., Tomlinson, A., Rao, A.,Coren, M., Winrow, A. and Britto, J. ISABEL: A web-based differentialdiagnostic aid for paediatrics: Results from an initial performanceevaluation. Archives of Disease in Childhood 88, 5 (2003), 408-413).

Autonomy Auminence uses the Autonomy technology to retrieve diagnosesgiven findings and organizes the diagnoses by body system(http://www.autonomyhealth.com). First CONSULT allows one to search alarge collection of medical books, journals, and guidelines by chiefcomplaints and age group to arrive at possible diagnoses(http://www.firstconsult.com). PEND DDX is a diagnosis generator basedon PEPID's independent clinical content(http://www.pepid.com/products/ddx/).

Clinical decision rules have been developed for a number of disorders,and computer systems have been developed to help practitioners andpatients apply these rules. The Acute Cardiac Ischemia Time-InsensitivePredictive Instrument (ACI-TIPI) takes clinical and ECG features asinput and produces probability of acute cardiac ischemia as output(Selker, H. P., Beshansky, J. R., Griffith, J. L., Aufderheide, T. P.,Ballin, D. S., Bernard, S. A., Crespo, S. G., Feldman, J. A., Fish, S.S., Gibler, W. B., Kiez, D. A., McNutt, R. A., Moulton, A. W., Ornato,J. P., Podrid, P. J., Pope, J. H., Salem, D. N., Sayre, M. R. andWoolard, R. H. Use of the acute cardiac ischemia time-insensitivepredictive instrument (ACI-TIPI) to assist with triage of patients withchest pain or other symptoms suggestive of acute cardiac ischemia: Amulticenter, controlled clinical trial. Annals of Internal Medicine129,11 (1998), 845-855). For example, ACI-TIPI is incorporated intocommercial heart monitors/defibrillators.

The CaseWalker system uses a four-item questionnaire to diagnose majordepressive disorder (Cannon, D. S. and Allen, S. N. A comparison of theeffects of computer and manual reminders on compliance with a mentalhealth clinical practice guideline. Journal of the American MedicalInformatics Association 7, 2 (2000), 196-203). The PKC Advisor providesguidance on 98 patient problems such as abdominal pain and vomiting(http://www.pkc.com/software/advisor/).

The strengths of current diagnosis systems are that they can improveclinicians' diagnostic hypotheses (Friedman, C. P., Elstein, A. S.,Wolf, F. M., Murphy, G. C., Franz, T. M., Heckerling, P. S., Fine, P.L., Miller, T. M. and Abraham, V. Enhancement of clinicians' diagnosticreasoning by computer-based consultation: A multisite study of 2systems. JAMA 282, 19 (1999), 1851-1856), and can help clinicians avoidmissing important diagnoses (Ramnarayan, P., Roberts, G. C., Coren, M.,Nanduri, V., Tomlinson, A., Taylor, P. M., Wyatt, J. C. and Britto, J.F. Assessment of the potential impact of a reminder system on thereduction of diagnostic errors: A quasi-experimental study. BMC Med.Inform. Decis. Mak. 6, 22 (2006)).

Current diagnosis systems are not widely used (Berner, E. S. DiagnosticDecision Support Systems: Why aren't they used more and what can we doabout it? AMIA Annu. Symp. Proc. 2006 (2006), 1167-1168, hereinafterreferred to as Berner, 2006) because the systems suffer from limitationsthat prevent them from being integrated into the day-to-day operationsof health organizations (Coiera, E. Guide to Health Informatics (SecondEdition). Hodder Arnold, 2003; and Shortliffe, T. Medical thinking: Whatshould we do? In Proceedings of Medical Thinking: What Do We Know? AReview Meeting (2006),http://www.openclinical.org/medicalThinking2006Summary2.html,hereinafter referred to as Shortliffe, 2006).

Many different healthcare workers may see a patient, and patient datamay be scattered across many different computer systems in bothstructured and unstructured form. Also, the systems are difficult tointeract with (Berner, 2006; Shortliffe, 2006). The entry of patientdata is difficult, the list of diagnostic suggestions may be too long,and the reasoning behind diagnostic suggestions is not alwaystransparent. Further, the systems are not focused enough on nextactions, and do not help the clinician figure out what to do to help thepatient (Shortliffe, 2006). The systems are also unable to ask thepractitioner for missing information that would increase confidence in adiagnosis, and they are not always based on the latest, high-qualitymedical evidence and have difficulty staying up-to-date (Sim, I.,Gorman, P., Greenes, R. A., Haynes, R. B., Kaplan, B., Lehmann, H. andTang, P. C. Clinical decision support systems for the practice ofevidence-based medicine. J. Am. Med. Inform. Assoc. 8, 6 (2001),527-534).

In view of these issues, the disclosed embodiments herein provide animproved medical diagnosis system.

SUMMARY

One exemplary method embodiment herein provides a decision-supportsystem for medical diagnosis and treatment. The system comprisessoftware modules embodied on a computer readable medium, and thesoftware modules comprise an input/output module and aquestion-answering module. The method receives patient case informationusing the input/output module and generates a medical diagnosis ortreatment query based on the patient case information and also generatesa plurality of medical diagnosis or treatment answers for the queryusing the question-answering module. The method also calculatesnumerical values for multiple medical evidence dimensions from medicalevidence sources for each of the answers using the question-answeringmodule, and also calculates a corresponding confidence value for each ofthe answers based on the numerical value of each evidence dimensionusing the question-answering module. The method further outputs themedical diagnosis or treatment answers, the corresponding confidencevalues, and the numerical values of each medical evidence dimension forone or more selected medical diagnosis or treatment answers using theinput/output module.

An exemplary system embodiment system comprises a first repositorymaintaining patient case information, a computer processor operativelyconnected to the first repository, and a second repository operativelyconnected to the computer processor. The computer processor isconfigured to receive the patient case information from the firstrepository, to generate a medical diagnosis or treatment query based onthe patient case information, and to generate a plurality of medicaldiagnosis or treatment answers for the query. The computer processor isalso configured to calculate numerical values for multiple medicalevidence dimensions from medical evidence sources for each of theanswers and to calculate corresponding confidence values for each of theanswers based on the numerical values of each medical evidencedimension. The computer processor is further configured to output themedical queries, the medical answers, the corresponding confidencevalues, and the numerical values of each medical evidence dimension tothe second repository.

An additional embodiment herein comprises a computer program productcomprising a computer readable storage medium storing computer readableprogram code comprising instructions executable by a computerizeddevice. The computer program code comprises an input/output modulereceiving patient case information, a patient case analysis moduleanalyzing the patient case information in order to identify semanticconcepts, a question generation module generating a medical diagnosis ortreatment query from the semantic concepts, and a question-answeringmodule generating a plurality of medical diagnosis or treatment answersfor the query. The question-answering module calculates numerical valuesfor multiple medical evidence dimensions from medical evidence sourcesfor each of the answers, calculates corresponding confidence values foreach of the answers based on the numerical value of each medicalevidence dimension using the question-answering module. Also, theinput/output module outputs the medical diagnosis or treatment queries,the medical diagnosis or treatment answers, the corresponding confidencevalues, and the numerical values for multiple medical evidencedimensions.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a system architecture chartfor an embodiment herein;

FIG. 2 is a schematic diagram illustrating of the decision supportprocess flow;

FIG. 3 is a schematic diagram illustrating a semantic model for themedical domain;

FIG. 4 is a schematic diagram illustrating the marginal contribution ofevidence along the dimensions of present illness, family history,findings, and demographics for four disease answers;

FIG. 5 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 6 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 7 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 8 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 9 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 10 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 11 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 12 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 13 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 14 is a schematic diagram illustrating an embodiment herein appliedto the medical domain;

FIG. 15 is a schematic diagram illustrating a computing node accordingto an embodiment herein;

FIG. 16 is a schematic diagram illustrating a cloud computingenvironment according to an embodiment herein; and

FIG. 17 is a schematic diagram illustrating an abstraction model layersaccording to an embodiment herein.

DETAILED DESCRIPTION

The following disclosure explains a decision-support application forproblem solving in a particular domain. The domain can be specific, forexample differential diagnosis in the medical domain, as will bediscussed below, or broader ranging. The objective of thedecision-support application is to inform a problem solving processbased on relevant contextual information for the problem, as describedin a target case. This case input information can be structured,unstructured or in other forms. Decision support is provided using aquestion-answering system that takes in questions or queries and returnsa list of answers and associated confidences.

When the method refers to a question-answering system, it means a systemthat can take an input query expressed in many possible forms, includingnatural language, structured language or many other means. Note, that ananswer need not be limited to be a single “atomic” concept (such asperson, place or thing), but may be a complex entity. Some examples ofcomplex entities are elaborate explanations reached through complexreasoning or a sequence of process steps required to achieve theintended goal of the user. Embodiments herein can be applied to avariety of domains involving complex systems that require human expertsto solve problems. For example, a detailed explanation is provided for adecision support application targeting differential diagnosis andtreatment in the medical domain (as an example of one of many domains).As would be understood by those ordinarily skilled in the art, thissystem can be used for other complex systems as well.

One embodiment herein allows “mixed-initiative” dialog. For example, theuser may ask queries and get answers from the application. Additionally,the application can automatically provide “push” notifications (e.g.,alerting of some significant change) to the user or ask queries of theuser that would help change the systems confidence in answers that itprovides. In effect, the system can continuously monitor relevant caseinput as well as take directed queries regarding a specific case.

FIG. 1 is a schematic diagram illustrating a broader decision-makingcontext in a system architecture chart of an embodiment herein. Thedecision-maker 108 may enter information through a range of devicesincluding mobile phones, tablets, computers, appliances, etc. Thisinformation can be input through a variety of modalities includingspoken, typed, constructed through a series of GUI interactions, etc.The information can be either problem case information or a query. Thequery can be in the form of natural language, structured language, orany other query format. The problem case information that the systemuses can be multimodal and can take the form of text, images, audio, orany other media form.

In general, the embodiments herein are intended to allow interaction tooccur over a period of time and to support an iterative refinementapproach. Therefore, one aspect of embodiments herein is a repository ofall relevant analysis and decisions made to date. This repository 106contains a representation of the reasoning and decision process not onlyas an efficiency mechanism, but allows the system to re-evaluateassumptions and decisions in light of new evidence relevant to the case.This allows users to interact with this representation, accepting,rejecting, or modifying it to as they think necessary to explorealternative solutions based on the users' insights into the validity orimportance of the evidence or reasoning chain. This repository 106 isnot only useful in the current evolving decision making interaction, butcan be used to track the provenance of decisions that were made in thepast and allow notification of actions to take based on newly arrivinginformation that comes possibly years after decisions were made. Forexample, if a new study reports a contraindication for a drug in a givensituation, the system could use this repository 106 of prior analysis toreevaluate its conclusions and provide relevant notification ofalternative therapies to a patient that has been on this drug for years.

Lastly, in general, all embodiments herein are meant to inform thedecision making process and allow the decision-maker 108 to viewalternatives and associated confidences in proposed answers, explore theevidence and reasoning process the system used to come to itsconclusions, and to get feedback on what additional information, ifprovided, would result in changing the answers.

The term diagnosis used in the medical domain can be generalized to mean“inform” in other domains. The medical examples found herein illustratesthis through answers, confidences, dimensions of evidence, associatedevidence passages, and documents where this evidence is found, as wellas, reliability of the evidence source. In the medical domain, theembodiments herein can be used as a clinical decision support tool byphysicians who are providing care to a patient. Examples of queriesinclude (but are not limited to): what clinical conditions arecharacterized by a set of symptoms?; what is the “differentialdiagnosis” (a ranked list of diseases) that could potentially cause aset of symptoms, conditions, findings? (this can be conditioned byproviding other pertinent patient information such as active diseases,current medications, allergies, past disease history, family diseasehistory and patient demographics); what tests would increase or decreaseconfidence in a given disease hypothesis present in the differentialdiagnosis?; and/or what treatments are recommended for a specifieddisease, given information about the patient?; etc. In the medicaldomain, the problem case information can be electronic medical records.

The question-answering system derives answers from a repository of‘domain knowledge’. The embodiments herein leave delineation of thedomain knowledge up to the question-answering system. Thequestion-answering system in the exemplary medical implementationactually uses the natural language of medical text books, clinicalguidelines, and other documents as the domain knowledge as well asstructured sources of information provided in databases or ontologies orany other potential structured form.

In general, the embodiments herein describe a decision-supportapplication 104 that is positioned in-between a source of problem caseinformation 102 and a question-answering system 110 using the example ofa medical diagnosis system. However, as would be understood by thoseordinarily skilled in the art, the embodiments herein are not limited tomedical diagnosis systems. To the contrary, the embodiments herein applyto diagnostic problem solving in any other complex-system domains thatrequire question answering over unstructured material. Examples of suchdomains include aircraft maintenance (and other vehicles of similarcomplexity) and information technology support (call centers). Thissystem could be used for automobiles, submarines, or even less complexbut likely more universally accessible applications, for example,finding answers to “how to” queries in information technology orsuitable meal recipes given input specifications such as ingredients,cost requirements, time, complexity, elegance, etc. These are allcharacterized by large amounts of structured and unstructured “problemcase information” and domain knowledge, and requiring a deepquestion-answer type technology. Therefore, while the examples hereinutilize a medical diagnosis system, those ordinarily skilled in the artwould understand that the embodiments herein are applicable to allquestion-answer systems and the embodiments herein are not limited tojust the exemplary medical diagnosis system that is used as a platformto illustrate the embodiments herein.

The system 104 comprises software modules (embodied on a computerreadable medium) including an input/output module, a problem caseanalysis module, a question generation module, a hypotheses and evidencemodule, etc. The QA system 110 includes a question-answering module,etc. The objective of decision-making is to diagnose and solve problemsthat arise in a complex system 100 specified in the domain. A humanexpert (user) 108 who interacts with the decision-support application104 (through items 100 and 102) makes decisions. A record of pastdecisions and the associated information used to arrive at the decisionis maintained in the repository 106. In some embodiments, thequestion-answering module can match the query to at least one of thepreviously generated medical diagnosis queries that were generated bythe question generation module and stored in the repository 106.

The decision-support application 104 may be triggered in several ways.In one mode, the decision-maker 108 asks a query about a particularcase. The application 104 may expand the query with relevant problemcase information and submit it to the question-answering (QA) system110. The resulting answers or hypotheses are then presented to thedecision-maker 108 who may iterate further, honing in to an acceptableresolution of the problem.

Another mode of operation assumes the existence of standing queriesdefined by the question generation module and/or the decision-maker 108.As new case information comes in, these queries are automatically run bythe decision-support application 104 without the active involvement ofthe decision-maker 108. The results may be proactively sent to thedecision-maker 108 or stored in the repository 106 for a subsequentscheduled interaction.

The above modes of operation assume the presence of active problem casesthat are yet to be satisfactorily resolved. Another mode of operationcan be triggered by changes in the content of domain knowledge used bythe question-answering system in item 112. In the medical domain, newclinical literature and guidelines are continuously being published,describing new screening procedures, therapies, and treatmentcomplications. The decision-support application 104 can use itsrepository 106 of past analyses and decisions to determine if any of itsprevious cases would be sufficiently affected by this new knowledge, andif so, send alerts to the responsible decision-makers 108.

Thus, FIG. 1 can also be considered to illustrate an exemplary systemembodiment herein that comprises a first repository 102 maintainingproblem case information, a computer processor 104/110 (running themodules) operatively connected to the first repository 102, and a secondrepository 106 operatively connected to the computer processor 104/110.Items are considered operatively connected to each other when the itemsare directly or indirectly connected to one another (e.g., physically,functionally, wired, wirelessly, etc.).

This “computer processor” 104/110 automatically analyzes the problemcase information in order to identify semantic concepts, relations, anddata and automatically generates at least one diagnosis query from thesemantic concepts, relations and data. The computer processor 104/110also automatically generates a plurality of diagnosis answers for eachdiagnosis query, and calculates confidence values for each of theanswers based on numerical values for several dimensions of evidencethat are relevant to the problem-solving domain. The computer processor104/110 can then automatically calculate corresponding confidence valuesfor each of the diagnosis answers based on the numerical value of eachevidence dimension of evidence sources of the confidence values.Further, the computer processor 104/110 can then automatically generatelinks to each item of evidence to allow the user to examine the passagesthat justify the answer in a given. For example, links can be generatedto the source of a passage, its type (text book, guideline, journalarticle, web content, database or structured source). The computerprocessor 104/110 also outputs the queries, the answers, thecorresponding confidence values, the links to the evidence sources, andthe numerical value of each evidence dimension to the decision-maker 108and/or the second repository 106.

FIG. 2 is a schematic diagram illustrating the decision support processflow performed by the decision support application 104. Morespecifically, FIG. 2 illustrates the flow of information from theproblem case information 102 to the question-answering system 110 andthe flow of information returned from the question-answering system 110back to the decision-maker 108. Thus, the method receives problem caseinformation using the input/output module. Further, in the medicaldomain, the problem case information can comprise illness symptoms of apatient, family history of the patient, demographics of the patient,etc. The output from the question-answering system 110 is used by thedecision-maker 108 to either make a decision or seek additionalinformation about the problem.

In item 202, the method receives input about the current problem. Themethod can receive a user inquiry through the input/output module in theform of a free-form query, a free-form statement, and/or keyword search,etc. The input from the problem can be multi-modal, such as text, audio,images, and video. The text can be unstructured, such as paragraphs ofproblem description in natural language, or structured, such as thecontent derived from a database. For example, in the medical domain, theinput can be clinical information pertinent to a patient's “History ofPresent Illness” (HPI). This can be in the form of paragraphs ofunstructured text describing any aspect of the patient's HPI as writtenor dictated by a nurse or physician, or semi-structured, with shortersentences or snippets assigned to specific HPI categories.

The input information can come in over time. The input may be triggeredby a change in the problem condition, the result of additional tests orprocedures performed, or a response to a query for more informationgenerated by the decision-maker 108. In addition, the information withinthe domain knowledge content 102 can change according to evolvingdemographic changes, evolving medical discoveries, evolving medicationconflicts, evolving side effect information, etc. This time-stampedinformation is recorded in the repository 106 in the system.

In item 204, the method automatically analyzes the problem caseinformation 102, using the problem case analysis module, in order toidentify semantic concepts, relations and other relevant knowledge(e.g., medical patient data). Thus, the method identifies semanticconcepts, relations and other relevant knowledge when the incominginformation is unstructured, such as natural language text, audio orimages, the concepts, relations and other kinds of information relevantto the domain has to be identified. This is done by software componentscalled “annotators”. They can be procedural code, rule based, usingprogrammed logic or in many other forms for determining concepts,relations and other relevant information. They could, for example, bebased on machine learning, using a set of training data containing knownconcepts and relations.

For the medical domain, annotators can recognize the phrases relating toclinical concepts such as patient symptoms, current medical conditions,clinical findings, medications, family history, demographics, etc.Annotators may also identify relations between entities such as locationof symptom, the severity of a condition, or the numerical value of afinding. The concepts and relations are represented by domain-specificsemantic model or type system. An example of such a semantic model forthe medical domain is shown in FIG. 3. More specifically, in the exampleshown in FIG. 3, various elements have different logical/causalrelationships. For example, substance 302 has an “is a” relationship toagent 304 indicating that substance 302 “is an” agent 304. Similarly, adisease/syndrome 306 can be caused by the agent 304, and thedisease/syndrome can be a complication of another disease/syndrome initem 306.

With respect to the disease/syndrome 306, it can be confirmed by test308, may be located at anatomy location 310, can be presented by afinding 318, and can be a complication of a certain treatment 312 (ormay be treated by the treatment 312). The treatments 312 may be aprocedure 314, a drug 316, etc. Similarly, with respect to the finding318, it may be measured by the test 308, located at the anatomy location310, treated by the treatment 312, be a side effect of the treatment312, or may specify a clinical attribute 320. Additionally, the clinicalattribute 320 may be influenced by the treatment 312. Therefore, thesemantic model illustrated in FIG. 3 (which may be referred to as afactoid physiology definition guideline) illustrates various conceptsand relations of a domain-specific semantic model.

In item 206, the method can receive queries or automatically generatequeries from the semantic concepts, relations and data using thequestion generation module. Thus, using the semantic concepts andrelations found in the previous step, queries for the question-answeringsystem can be automatically formulated. Alternatively, it is alsopossible for the decision-maker 108 to enter queries in natural languageor other ways, as described above.

In case of automatic formulation, a set of “standing” queries can bedesigned as a template. For example, a standing query in the medicaldomain is the “differential diagnosis.” This is a list of potentialhypotheses of the diseases or other medical conditions that explain apatient's symptoms and abnormal findings. The diagnosis query templatesherein have blank slots for concepts such as symptoms, findings, pastdiseases, current medications, allergies, etc. Once the semanticconcepts and relations are identified, these fill in the blanks in thetemplate, resulting in a synthesized query. The concept of a template isa general computational element for automatically constructing a set ofrelevant queries (queries) to the underlying question-answering systemthat is used to synthesize and return information relevant to thespecific information need at hand.

There are many ways to implement templates. For example, queries may beautomatically generated in item 206 based on what is known and unknownabout the problem case. For example, in the medical domain, if symptomand finding concepts have been identified in the patient caseinformation, but no diseases are found, a diagnosis query may begenerated. The physician is also able to type in a query such as “Whatis the diagnosis?” and rely on the rest of the context to come from thesemantic concepts. The physician is also able to fine-tune the queryingby specifying more constraints such as “Is there an infectious cause ofthese symptoms?”

In item 208, the method sends queries to the QA system 110. Thus, themethod can automatically generate a plurality of answers for each queryusing the question-answering module. Once a query is formulated, thequestion-answering system 110 is invoked. For aiding the subsequentinterpretation of the answers, a query may be converted into multiplequeries. Each query in this set may contain a subset of the conceptsfound about the problem. For example, a clinical diagnosis querycontaining symptoms, findings, family history and demographicinformation, could generate a series of queries as follows, where thetext in the <>characters is replaced by the corresponding concepts foundin the case text: “What disease of condition could cause <symptom>?”;“What disease of condition could cause <symptom>and <findings>?; “Whatdisease of condition could cause <symptom>, <findings>and <familyhistory>?; “What disease of condition could cause <symptom>, <findings>,<family history>and <demographics>?; etc. This build-up of informationin the query makes it possible to calculate the marginal contribution offindings, family history and demographic information to the confidenceof a diagnosis. Other strategies for breaking down a query into a set ofqueries could also be used.

The method receives answers from the question-answering system in item210. For each query submitted, the question-answering system 110 returnsa list of answers, their confidences, evidence dimensions, and evidencesources. The confidence of each answer can, for example, be a numberbetween 0 and 1. This confidence is constructed from various answerscorers in the question-answering system, which evaluates thecorrectness of the answer according to various dimensions of evidencesources. For example, a candidate answer to a medical diagnosis querycan be evaluated in terms of the semantic type of the answer. The scorealong this dimension will be high if the answer is a type of disease ormedical condition that can be considered as a diagnosis. For everyanswer to a query, the passages of domain knowledge from which theanswer was extracted are also available from the question-answeringsystem. This can be snippets of text that match the structure of thequery or entire documents that were returned from search componentsduring the question-answering process. For each passage, a citation tothe original source of information is also recorded.

In item 212, the method further automatically calculates confidencevalues for each of the answers based on numerical values for severaldimensions of evidence that are relevant to the problem-solving domain.The numerical value of each evidence dimension can be based upon thevarious semantic concepts and relations found in the problem caseinformation 102, as described by the method in item 204. For example, inthe medical domain, these could be the patient's symptoms, findings,family history, demographics, etc.

The above processes described methods of formulating multiple queriescontaining a subset of the concepts found in the problem text. Byanalyzing answers and their confidences for these queries, an estimateof the marginal contribution of these concepts can be generated. For theexample for the queries generated, the marginal impact of symptoms,findings, family history and demographics are calculated. Othertechniques for achieving this are possible as well.

In item 214, the method displays information to support decision-making.The list of answers is displayed along with answer confidences for thedecision-maker 108 to evaluate (see FIG. 4 for an example). Thus, themethod outputs the queries, the answers, the corresponding confidencevalues, the links to the evidence sources, and the numerical value ofeach evidence dimension using the input/output module upon user inquiry.Additionally, the decision maker can explore each evidence dimensionfurther by viewing each piece of evidence and explore its associatedprovenance. For example, a piece of evidence may be a supportingpassage, reasoning chain, or database fact. Similarly, examples ofassociated provenances include journal articles, textbooks, anddatabases. Further, when outputting the numerical value of each evidencedimension , this embodiment can illustrate the amount each evidencedimension contributes to a corresponding confidence value (on a scale orpercentage basis, for example) and illustrate how changes in each of thenumerical value of each evidence dimension produce changes in thecorresponding confidence value.

Further, the embodiments herein automatically and continuously updatethe diagnosis answers, the corresponding confidence values, and thenumerical value of each evidence dimension based on revisions to theproblem case information to produce revised queries, answers,corresponding confidence values, etc. (using the question-answeringmodule). This method can also automatically output the revised queries,answers, and/or corresponding confidence values when a differencethreshold is exceeded. This “difference threshold” can comprise a timeperiod (e.g., hours, weeks, months, etc.), the amount one or moreanswers change (e.g., percentage change, polarity (yes/no) change,number of answers changing, etc.) and/or an amount of confidence valuechanges (percent confidence change, confidence polarity change, etc.).

Therefore, the decision support application 104 continuously anddynamically automatically provides queries and answers based upon theevolving semantic concepts, relations and other relevant data (e.g.,medical patient data) in order to provide the highest confidence answersand the most information on such answers to the decision-maker 108.Rather than providing static applications that always provide the sameanswers when given the same input (as is done conventionally), theembodiments herein continually update the values and relationships ofthe numerical values of each evidence dimension to change the confidencevalues of potential answers. When the confidence values of the potentialanswers change, the answers that are most highly recommended can alsochange, thereby dynamically allowing the decision-maker to be providedwith different best answers as the problem case information evolves overtime.

Thus, the embodiments herein provide substantial advantages over systemsthat generate answers and confidence values based on preset, fixedcriteria that is rarely revised (or only revised at periodic updates(e.g., software updates)). For example, in the medical domain, by actingdynamically, previous answers and recommendations can change based onevolving demographic changes, evolving medical discoveries, evolvingmedication conflicts, evolving side effect information, etc., within thedomain knowledge content 112. Therefore, the embodiments herein canchange a course of medical treatment advice for a patient, even if thatpatient does not experience a personal change, merely because other datawithin the domain knowledge content 112 evolves over time. This allowsmedical providers a fully automated system for constantly prescribingthe best medical treatment for their patients as medical advances anddemographics change over time.

In many domains, the answer with the highest confidence need not be theappropriate answer because there can be several possible explanationsfor a problem. For example, in the medical domain, several diseases maycause a patient to present a set of symptoms. In addition to displayinga list of answers and their confidences, one or more of the answers maybe selected to drill down into the dimensions of evidence. FIG. 4 is aschematic diagram illustrating the contribution of each dimension valueof evidence from evidence sources to the overall confidence of ananswer. The output shown in FIG. 4 compares each dimension acrossmultiple answers. FIG. 4 illustrates the marginal contribution ofevidence 400 along the dimensions of present illness 402, family history404, findings 406, and demographics 408 for four disease answers. Inthis example, the “dimensions” are ‘present illness’, ‘findings’,‘family history’, and ‘demographics’ and each has its own value. Thiscomparative analysis of multiple answers along the evidence dimensionsallows the decision-maker 108 to consider and visualize the trade-offsin evidence in order to arrive at a decision.

The decision-maker 108 can also drill down deeper into each answer anddimension of evidence and examine the supporting pieces of evidence thatjustify the answer along that dimension. For example, the source of thepassage, its type (text book, guideline, journal article, web content)and a link to the source is provided for the decision maker to delvedeeper and confirm its validity.

The method can also identify missing information in item 216. Morespecifically, this embodiment automatically identifies informationrelevant to the answers that is not contained within the problem caseinformation as missing information, and further automatically identifiesthe amount the missing information affects the corresponding confidencevalues (both using the using the question-answering module) and outputsthis information to the user.

If the answers and their evidence returned by the question-answeringsystem are not adequate for arriving at a decision, the application 104may be used to identify missing information that has potential foraffecting the confidence in answers. For a given answer, thedecision-maker 108 may want to know what hypothetical information, ifprovided, can produce the greatest change in the confidence. Forexample, in the medical domain, if the answer is a disease, the missinginformation may be a lab test that confirms or rules out the disease. Itmay also be other signs or symptoms not specified for the patient. Inreality, there may be a large amount of missing information associatedwith an answer and the embodiments herein can rank the missinginformation. Characteristics that can be used to rank the potentialvalue of the missing information are factors such as the cost ofobtaining this information, the time taken, and the amount by which themissing information affects the confidence of the answer.

When two answers have similar confidences, making it difficult to choosebetween them, it is helpful to identify the missing information thatwill cause the biggest difference between these confidences. Forexample, in the medical domain, the answers may be two related diseasesand the missing information may be a lab test designed to differentiatebetween them. This evidence could increase as well as decrease theconfidence of one answer thus helping to ascertain the correct diagnosisin the case of a medical diagnostic system.

The identification of missing information need not only be done at theinitiative of the decision-maker 108. When certain criteria are met, forexample, confidence of two top answers are very close, the application104 itself may take the initiative and may automatically request themissing information.

Once the missing information is identified, the decision-maker 108 hasto seek this missing information using procedures specific to thedomain. In the medical domain, this may require ordering lab tests orasking the patient for more information. When this missing informationbecomes available, it is sent back to the decision-support as describedabove and a new iteration of question-answering and decision supportprocess illustrated in FIG. 2 is begun.

FIGS. 5-14 are schematic diagrams of screenshots that can be presentedto the users. In FIG. 5, profiles 502, 504 for two patients are shown ona screenshot 500. Additional profiles for further patients can becreated. In FIG. 6, the first patient 502 has been selected and hasdescribed symptoms that are listed in the History of Present Illnesssection 506. This information can be input by a health care professionalinto an Electronic Health Record (EHR) or merely made available for thesystem to consider by typing into the box 506. The proposed system canpull the relevant information automatically from the health record ortext field above, use analytics to find relevant concepts, classify themas belonging to the symptoms dimension and automatically generate thequery listed in the queries field 508. Alternatively, the user(physician/patient) could enter a query directly into the query field.The user can then click on the “Ask Watson” button 510 to proceed. FIG.6 also lists an “Evidence” button 512, which is discussed below.

In FIG. 7, the decision-support application 104 has generated a set ofpossible answers to the query with associated confidence scoresassociated with each answer and the same is displayed in area 514. InFIG. 8, having confirmed the condition, the user can enter the conditioninto the History of Present Illness section, or the condition can beextracted automatically from the EHR. Subsequently, the physician canask another query or have the decision-support application 104automatically generate another query.

In FIG. 9, the process continues with new information having been addedin item 506 that is analyzed and grouped in relevant dimensions such aspresent illness, family history, etc. In FIG. 10, the same processcontinues as more information continues to be added in item 506, therebyrefining the potential diagnosis. In FIG. 11, the process has reached apoint where the decision-support application 104 indicates in item 514with high confidence over other potential answers that the properdiagnosis for the particular patient is Lyme disease. From the exampleof FIG. 11, the information contained in these dimensions come from thecase information (respectively, ‘uveitis’, ‘circular rash . . . ’,‘arthritis’, and ‘Connecticut’). The numerical value of each evidencedimension comes from the presence of the information contained in thesedimensions in the medical content in the context of the hypothesizedanswer (e.g., Lyme disease).

In FIG. 12, the decision-support application 104 allows the user toselect the answer Lyme disease in order to view the evidence profile 516for the answer. The application 104 reveals the dimensions of evidenceand their associated contribution to the Lyme disease diagnosis. Theuser can then further select a particular dimension to explore snippetsof evidence that contribute to this dimension. Finally, the application104 allows for the physician to view the whole documents from which thesnippets were derived by clicking on one of the links labeled 518 inFIG. 12, such as a textbook, journal, or website. In FIG. 13, thedecision-support application 104 is again shown, except in this case theapplication 104 is directed towards exploring possible treatments fortreating the identified condition. In FIG. 14, new information has beenadded or automatically extracted from the patient's medical recordrelevant to the appropriate treatment to the identified condition. Inthis case, the application 104 has identified that the patient isallergic to penicillin and that the patient is pregnant. The application104 uses this information to find the appropriate treatment, in thiscase indicating a confidence score for a particular treatment option.

As will be appreciated by one skilled in the art, aspects of theembodiments herein may be embodied as a system, method or computerprogram product. Accordingly, aspects of the embodiments herein may takethe form of an entirely hardware embodiment, an entirely softwareembodiment (including firmware, resident software, micro-code, etc.) oran embodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the embodiments herein may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM),an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of theembodiments herein may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the embodiments herein are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks. The computer program instructions may also beloaded onto a computer, other programmable data processing apparatus, orother devices to cause a series of operational steps to be performed onthe computer, other programmable apparatus or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments herein. In this regard, each block in the flowchart or blockdiagrams may represent a module, segment, or portion of code, whichcomprises one or more executable instructions for implementing thespecified logical function(s). It should also be noted that, in somealternative implementations, the functions noted in the block may occurout of the order noted in the figures. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently, or theblocks may sometimes be executed in the reverse order, depending uponthe functionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

It is understood in advance that although this disclosure includes adetailed description on cloud computing, implementation of the teachingsrecited herein are not limited to a cloud computing environment. Rather,embodiments herein are capable of being implemented in conjunction withany other type of computing environment now known or later developed.Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g. networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services)that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models. Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service. Service Models are asfollows: Software as a Service (SaaS): the capability provided to theconsumer is to use the provider's applications running on a cloudinfrastructure. The applications are accessible from various clientdevices through a thin client interface such as a web browser (e.g.,web-based e-mail). The consumer does not manage or control theunderlying cloud infrastructure including network, servers, operatingsystems, storage, or even individual application capabilities, with thepossible exception of limited user-specific application configurationsettings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises. Community cloud: the cloudinfrastructure is shared by several organizations and supports aspecific community that has shared concerns (e.g., mission, securityrequirements, policy, and compliance considerations). It may be managedby the organizations or a third party and may exist on-premises oroff-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure comprising anetwork of interconnected nodes.

Referring now to FIG. 15, a schematic of an example of a cloud computingnode is shown. Cloud computing node 10 is only one example of a suitablecloud computing node and is not intended to suggest any limitation as tothe scope of use or functionality of embodiments of the inventiondescribed herein.

Regardless, cloud computing node 10 is capable of being implementedand/or performing any of the functionality set forth hereinabove. Incloud computing node 10 there is a computer system/server 12, which isoperational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system/server 12 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system/server 12 may be described in the general context ofcomputer system executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Computer system/server 12 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 15, computer system/server 12 in cloud computing node10 is shown in the form of a general-purpose computing device. Thecomponents of computer system/server 12 may include, but are not limitedto, one or more processors or processing units 16, a system memory 28,and a bus 18 that couples various system components including systemmemory 28 to processor 16. Bus 18 represents one or more of any ofseveral types of bus structures, including a memory bus or memorycontroller, a peripheral bus, an accelerated graphics port, and aprocessor or local bus using any of a variety of bus architectures. Byway of example, and not limitation, such architectures include IndustryStandard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus,Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA)local bus, and Peripheral Component Interconnects (PCI) bus. Computersystem/server 12 typically includes a variety of computer systemreadable media. Such media may be any available media that is accessibleby computer system/server 12, and it includes both volatile andnon-volatile media, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system/server 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,memory 28 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention. Program/utility 40, having aset (at least one) of program modules 42, may be stored in memory 28 byway of example, and not limitation, as well as an operating system, oneor more application programs, other program modules, and program data.Each of the operating system, one or more application programs, otherprogram modules, and program data or some combination thereof, mayinclude an implementation of a networking environment. Program modules42 generally carry out the functions and/or methodologies of embodimentsof the invention as described herein. Computer system/server 12 may alsocommunicate with one or more external devices 14 such as a keyboard, apointing device, a display 24, etc.; one or more devices that enable auser to interact with computer system/server 12; and/or any devices(e.g., network card, modem, etc.) that enable computer system/server 12to communicate with one or more other computing devices. Suchcommunication can occur via Input/Output (I/0) interfaces 22. Still yet,computer system/server 12 can communicate with one or more networks suchas a local area network (LAN), a general wide area network (WAN), and/ora public network (e.g., the Internet) via network adapter 20. Asdepicted, network adapter 20 communicates with the other components ofcomputer system/server 12 via bus 18. It should be understood thatalthough not shown, other hardware and/or software components can beused in conjunction with computer system/server 12. Examples, include,but are not limited to: microcode, device drivers, redundant processingunits, external disk drive arrays, RAID systems, tape drives, and dataarchival storage systems, etc.

Referring now to FIG. 16, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 comprises one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 2 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 17, a set of functional abstraction layersprovided by cloud computing environment 50 (FIG. 2) is shown. It shouldbe understood in advance that the components, layers, and functionsshown in FIG. 3 are intended to be illustrative only and embodiments ofthe invention are not limited thereto. As depicted, the following layersand corresponding functions are provided: Hardware and software layer 60includes hardware and software components. Examples of hardwarecomponents include mainframes, in one example IBM® zSeries® systems;RISC (Reduced Instruction Set Computer) architecture based servers, inone example IBM pSeries® systems; IBM xSeries® systems; IBM BladeCenter®systems; storage devices; networks and networking components. Examplesof software components include network application server software, inone example IBM WebSphere® application server software; and databasesoftware, in one example IBM DB2® database software. (IBM, zSeries,pSeries, xSeries, BladeCenter, WebSphere, and DB2 are trademarks ofInternational Business Machines Corporation registered in manyjurisdictions worldwide). Virtualization layer 62 provides anabstraction layer from which the following examples of virtual entitiesmay be provided: virtual servers; virtual storage; virtual networks,including virtual private networks; virtual applications and operatingsystems; and virtual clients. In one example, management layer 64 mayprovide the functions described below. Resource provisioning providesdynamic procurement of computing resources and other resources that areutilized to perform tasks within the cloud computing environment.Metering and Pricing provide cost tracking as resources are utilizedwithin the cloud computing environment, and billing or invoicing forconsumption of these resources. In one example, these resources maycomprise application software licenses. Security provides identityverification for cloud consumers and tasks, as well as protection fordata and other resources. User portal provides access to the cloudcomputing environment for consumers and system administrators. Servicelevel management provides cloud computing resource allocation andmanagement such that required service levels are met. Service LevelAgreement (SLA) planning and fulfillment provide pre-arrangement for,and procurement of, cloud computing resources for which a futurerequirement is anticipated in accordance with an SLA.

Workloads layer 66 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation; software development and lifecycle management; virtualclassroom education delivery; data analytics processing; transactionprocessing; and decision-support for problem solving using aquestion-answering system.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the embodiments herein has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method comprising: receiving, by adecision-support system for medical diagnosis and treatment, patientcase information regarding a patient, said decision-support systemcomprising a computerized device that has access to evidence sourcescontaining medical domain knowledge; analyzing, by said decision-supportsystem, said patient case information to identify semantic concepts byrecognizing phrases in said patient case information logically relatedto medical concepts; generating, by said decision-support system, aquery by automatically supplying said semantic concepts to blank slotsin a query template to generate said query, said query comprising amedical question; generating, by said decision-support system, answersto said query by applying natural language processing techniques tomatch said query to passages of said medical domain knowledge withinsaid evidence sources, said evidence sources comprising information inunstructured form; calculating, by said decision-support system,numerical values of evidence dimensions for each of said answers byassigning different numerical values to each of said answers based onthe correctness of each of said answers for different semantic concepts;calculating, by said decision-support system, corresponding confidencevalues for said answers by combining said numerical values of saidevidence dimensions as calculated for said answer; outputting, by saiddecision-support system to a decision-maker for said patient, saidanswers, said numerical values of said evidence dimensions for each ofsaid answers, and said corresponding confidence values for said answersso that said answers, said numerical values of said evidence dimensionsfor each of said answers, and said corresponding confidence values forsaid answers are usable by said decision-maker to make a decisionregarding said patient; storing, by said decision-support system in atangible repository, said answers, said numerical values of saidevidence dimensions for each of said answers, and said correspondingconfidence values for said answers; in response to new medical domainknowledge being added to said evidence sources, automatically repeating,by said decision-support system, said generating of said answers to saidquery, said calculating of said numerical values of said evidencedimensions for each of said answers and said calculating of saidcorresponding confidence values for said answers in order to determinewhether previously generated answers and corresponding confidence valuesstored in said tangible repository would change based on said newmedical domain knowledge; and, automatically sending, by saiddecision-support system, an alert to said decision-maker based on achange occurring in any of said previously generated answers andcorresponding confidence values, said alert being usable by saiddecision-maker to re-evaluate said decision.
 2. The method according toclaim 1, further comprising automatically repeating, by saiddecision-support system, said generating of said query, said generatingof said answers to said query, said calculating of said numerical valuesof said evidence dimensions for each of said answers, and saidcalculating of said corresponding confidence values for said answersbased on new patient case information being received.
 3. The methodaccording to claim 2, further comprising automatically sending, by saiddecision-support system, an additional alert to said decision-makerbased on any additional change occurring in any of said previouslygenerated answers and corresponding confidence values based on said newpatient case information being received.
 4. The method according toclaim 1, said generating of said answers being carried out by analyzingmedical domain knowledge content.
 5. The method according to claim 1,said generating of said query being carried out by receiving said queryin a form of at least one of a free-form query, a free-form statement,and keyword search.
 6. The method according to claim 5, furthercomprising receiving, by said decision-support system, an initial queryfrom said decision-maker, said generating of said query comprising: inresponse to said initial query, analyzing said patient case information,to identify said semantic concepts; and expanding said initial queryusing said semantic concepts to generate a specific natural languagequery.
 7. The method according to claim 1, further comprisingoutputting, for a selected evidence dimension, each piece of evidencesupporting said selected evidence dimension and an associated provenancefor said piece of evidence.
 8. The method according to claim 7, furthercomprising: removing, by said decision-support system, a selected pieceof evidence from said evidence sources contributing to a numerical valueof an evidence dimension of a specific answer; recalculating, by saiddecision-support system, said numerical value of said evidence dimensionof said specific answer having said selected piece of evidence removedto produce a recalculated numerical value; and recalculating, by saiddecision-support system, a new confidence value of said specific answerbased on said recalculated numerical value.
 9. The method according toclaim 1, further comprising: identifying, by said decision-supportsystem, information relevant to said answers that is not containedwithin said patient case information as missing information; andoutputting, by said decision-support system, a request to saiddecision-maker to add said missing information to said patient caseinformation.
 10. The method according to claim 9, further comprisingidentifying, by said decision-support system, an amount said missinginformation affects corresponding confidence values of said answers. 11.The method according to claim 1, said alert being sent only when saidchange exceeds a difference threshold, said difference threshold beingany of a percentage change in an answer, an answer polarity change, apercentage change in a confidence level and a confidence level polaritychange.
 12. A system comprising: a first tangible repository maintainingpatient case information regarding a patient; a computer processoroperatively connected to said first tangible repository and havingaccess to evidence sources containing medical domain knowledge; and asecond tangible repository operatively connected to said computerprocessor, said computer processor receiving said patient caseinformation from said first tangible repository, said computer processoranalyzing said patient case information to identify semantic concepts byrecognizing phrases in said patient case information logically relatedto medical concepts, and generating a query by automatically supplyingsaid semantic concepts to blank slots in a query template to generatesaid query, said query comprising a medical question, said computerprocessor generating answers to said query by applying natural languageprocessing techniques to match said query to passages of said medicaldomain knowledge within said evidence sources, said evidence sourcescomprising information in unstructured form, said computer processorcalculating numerical values of evidence dimensions for each of saidanswers by assigning different numerical values to each of said answersbased on the correctness of each of said answers for different semanticconcepts, said computer processor calculating corresponding confidencevalues for said answers by combining said numerical values of saidevidence dimensions as calculated for said answer, said computerprocessor outputting, to a decision-maker for said patient, said query,said answers, said numerical values of said evidence dimensions for eachof said answers, and said corresponding confidence values for saidanswers so that said answers, said numerical values of said evidencedimensions for each of said answers, and said corresponding confidencevalues for said answers are usable by said decision-maker to make adecision regarding said patient, said computer processor furtherstoring, in said second tangible repository, said answers, saidnumerical values of said evidence dimensions for each of said answers,and said corresponding confidence values for said answers, said computerprocessor further, in response to new medical domain knowledge beingadded to said evidence sources, automatically repeating said generatingof said answers to said query, said calculating of said numerical valuesof said evidence dimensions for each of said answers and saidcalculating of said corresponding confidence values for said answers inorder to determine whether previously generated answers andcorresponding confidence values stored in said second tangiblerepository would change based on said new medical domain knowledge, andsaid computer processor automatically sending an alert to saiddecision-maker based on a change occurring in any of said previouslygenerated answers and corresponding confidence values, said alert beingusable by said decision-maker to re-evaluate said decision.
 13. Thesystem according to claim 12, said computer processor furtherautomatically repeating said generating of said query, said generatingof said answers to said query, said calculating of said numerical valuesof said evidence dimensions for each of said answers, and saidcalculating of said corresponding confidence values for said answersbased on new patient case information being received.
 14. The systemaccording to claim 12, further comprising a third tangible repositoryfor maintaining medical domain knowledge content, said computerprocessor generating said answers by analyzing said medical domainknowledge content.
 15. The system according to claim 12, said computerprocessor generating said query being carried out by said computerprocessor receiving said query as an input in a form of at least one ofa free-form query, a free-form statement, and keyword search.
 16. Thesystem according to claim 12, said computer processor outputting a pieceof evidence supporting a selected evidence dimension and an associatedprovenance for said piece of evidence.
 17. The system according to claim16, said computer processor removing a selected piece of evidence fromsaid evidence sources contributing to a numerical value of an evidencedimension of a specific answer, recalculating said numerical value ofsaid evidence dimension of said specific answer having said selectedpiece of evidence removed to produce a recalculated numerical value, andrecalculating a new confidence value of said specific answer based onsaid recalculated numerical value.
 18. The system according to claim 12,said computer processor identifying information relevant to said answersthat is not contained within said patient case information as missinginformation and outputting a request to said decision-maker to add saidmissing information to said patient case information.
 19. The systemaccording to claim 18, said computer processor identifying an amountsaid missing information affects corresponding confidence values of saidanswers.
 20. A computer program product comprising a tangible computerreadable storage device storing computer readable program codecomprising instructions executable by a computerized device, saidcomputerized device having access to evidence sources containing medicaldomain knowledge and said computer program code comprising: aninput/output module receiving patient case information regarding apatient; a patient case analysis module analyzing said patient caseinformation to identify semantic concepts by recognizing phrases in saidpatient case information logically related to medical concepts; aquestion generation module generating a query by automatically supplyingsaid semantic concepts to blank slots in a query template to generatesaid query, said query comprising a medical question; and aquestion-answering module generating answers to said query by applyingnatural language processing techniques to match said query to passagesof said medical domain knowledge within said evidence sources, saidevidence sources comprising information in unstructured form, saidquestion-answering module calculating numerical values of evidencedimensions for each of said answers by assigning different numericalvalues to each of said answers based on the correctness of each of saidanswers for different semantic concepts, said question-answering modulecalculating corresponding confidence values for said answers bycombining said numerical values of said evidence dimensions ascalculated for said answer, said input/output module outputting, to adecision-maker for said patient, said query, said answers, saidnumerical values of said evidence dimensions for each of said answers,and said corresponding confidence values for said answers so that saidanswers, said numerical values of said evidence dimensions for each ofsaid answers, and said corresponding confidence values for said answersare usable by said decision-maker to make a decision regarding saidpatient, said input/output module further storing, said answers, saidnumerical values of said evidence dimensions for each of said answersand said corresponding confidence values for said answers in a tangiblerepository, said question-answering module further, in response to newmedical domain knowledge being added to said evidence sources,automatically repeating said generating of said answers, saidcalculating of said numerical values of said evidence dimensions foreach of said answers and said calculating of said correspondingconfidence values for said answers to determine whether previouslygenerated answers and corresponding confidence values stored in saidtangible repository would change based on said new medical domainknowledge, and said input/output module automatically sending an alertto said decision-maker based on a change occurring in any of saidpreviously generated answers and corresponding confidence, said alertbeing usable by said decision-maker to re-evaluate said decision.